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Linear Algebra In A Nutshell 685 



LINEAR ALGEBRA IN A NUTSHELL 



One question always comes on the first day of class. "Do / have to know linear 
algebra?'^ My reply gets shorter every year: "Ybw soon will.^^ This section brings 
together many important points in the theory. It serves as a quick primer, not an 
ofhcial part of the applied mathematics course (like Chapter 1 and 2). 

This summary begins with two lists that use most of the key words of linear 
algebra. The first list applies to invertible matrices. That property is described in 14 
different ways. The second list shows the contrast, when A is singular (not invertible). 
There are more ways to test invertibility of an n by n matrix than I expected. 



Nonsingular 

A is invertible 

The columns are independent 
The rows are independent 
The determinant is not zero 
Ax = has one solution j; = 
Ax = b has one solution x = A^^b 
A has n (nonzero) pivots 
A has full rank 

The reduced row echelon form is R 
The column space is all of R" 
The row space is all of R" 
All eigenvalues are nonzero 
A^A is symmetric positive definite 
A has n (positive) singular values 



Singular 

A is not invertible 

The columns are dependent 

The rows are dependent 

The determinant is zero 

Ax = has infinitely many solutions 

Ax = b has no solution or infinitely many 

A has r < n pivots 

A has rank r < n 

R has at least one zero row 

The column space has dimension r < n 

The row space has dimension r < n 

Zero is an eigenvalue of A 

A'^A is only semidefinite 

A has r < n singular values 



Now we take a deeper look at linear equations, without proving every statement 
we make. The goal is to discover what Ax = b really means. One reference is my 
textbook Introduction to Linear Algebra, published by Wellesley-Cambridge Press. 
That book has a much more careful development with many examples (you could look 
at the course page, with videos of the lectures, on ocw.mit.edu or web.mit.edu/18.06). 

The key is to think of every multiplication Ax, a matrix A times a vector X , as a 
combination of the columns of A: 

Matrix Multiplication by Columns 



combination of columns . 



Multiplying by rows, the first component C + 2D comes from 1 and 2 in the first row 
of A. But I strongly recommend to think of Ax a column at a time. Notice how 
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X = (1, 0) and x = (0, 1) will pick out single columns of A: 

= last column . 

Suppose A is an m by n matrix. Then Ax = has at least one solution, the all-zeros 
vector X = 0. There are certainly other solutions in case n > m (more unknowns 
than equations). Even if m = n, there might be nonzero solutions to Ax = 0; then 
A is square but not invertible. It is the number r of independent rows and columns 
that counts. That number r is the rank of A [r < m and r <n). 
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The nuUspace of A is the set of all solutions x to Ax = 0. This nuUspace 
N{A) contains only x = when the columns of A are independent. In 
that case the matrix A has full column rank r = n: independent columns. 



For our 2 by 2 example, the combination with C = 2 and D = —1 produces the zero 
vector. Thus x = (2,-1) is in the nullspace, with Ax = 0. The columns (1,3) and 
(2, 6) are "linearly dependent." One column is a multiple of the other column. The 
rank is r = 1. The matrix A has a whole line of vectors cx = c(2, —1) in its nullspace: 



Nullspace 
is a line 
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If Ax = and Ay = 0, then every combination cx + dy is in the nullspace. Always 
Ax = asks for a combination of the columns of A that produces the zero vector: 



X in nullspace 



^1 (column 1) 



Xn (column n)= zero vector 



When those columns are independent, the only way to produce Ax = is with 2:1 = 0, 
X2 = 0, . . ., Xn = 0. Then x = (0, ... ,0) is the only vector in the nullspace of A. 
Often this will be our requirement (independent columns) for a good matrix A. In 
that case, A^A also has independent columns. The square n by n matrix A^A is then 
invertible and symmetric and positive definite. If A is good then A'^A is even better. 

I will extend this review [still optional) to the geometry of Ax = b. 



Column Space and Solutions to Linear Equations 

Ax = b asks for a linear combination of the columns that equals 6. In our 

2 by 2 example, the columns go in the same direction! Then b does too: 



Column space 



Ax 
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is always on the line through 



We can only solve Ax = b when the vector b is on that line. For b = (1,4) there is no 
solution, it is off the line. For b = (5, 15) there are many solutions (5 times column 1 
gives b, and this b is on the line). The big step is to look at a space of vectors: 
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Definition: The column space contains all combinations of the columns. 

In other words, C{A) contains all possible products A times x. Therefore Ax = 6 is 
solvable exactly when the vector b is in the column space C{A). 

For an m by n matrix, the columns have m components. The column space of 
A is in m-dimensional space. The word "space" indicates that the key operation of 
linear algebra is allowed: Any combination of vectors m the space stays in the space. 
The zero combination is allowed, so the vector x = is in every space. 

How do we write down all solutions, when b belongs to the column space of A? 
Any one solution to Ax = 6 is a particular solution Xp. Any vector Xn in the 
nullspace solves Ax = 0. Adding Axp = b to Axn = g b. The 

complete solution to Ax = b has this form 



Complete solution 
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In the example, b = (5, 15) is 5 times the first column, so one particular solution 
is Xp = (5, 0). To find all other solutions, add to Xp any vector j;„ in the nullspace — 
which is the line through (2, —1). Here is Xp + (all a;„): 

gives 2;(,Qj^p]g^g 



"12' 




■ c ■ 




5 " 


3 6 




D 




15 



C 




5 




2c 


D 







+ 


— c 



This line of solutions is drawn in Figure Al. It is not a subspace. It does not 
contain (0,0), because it is shifted over by the particular solution (5,0). We only 
have a "space" of solutions when b is zero (then the solutions fill the nullspace). 




shortest solution pinv(A) * & is in the row space 
line of all solutions Xp + all .x„ (not a subspace) 

[q] = one particular solution Xp = A\b 

C 



: nullspace 



Figure Al: Parallel lines of solutions to Axn = and [3 g] [xp + x^) = {^^] 



May I collect three important comments on linear equations Ax = b. 

1. Suppose A is a square invertible matrix (the most common case in practice). 
Then the nullspace only contains Xn = 0. The particular solution Xp = A^^b is 
the only solution. The complete solution Xp I IS 

A-^b + 0. Thus X = A-^b. 
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2. Ax = b has infinitely many solutions in Figure Al. The shortest x always lies 
in the "row space" of A. That particular solution (1, 2) is found by the pseudo- 
inverse pinv (A). The backslash A\b finds an x with at most m nonzeros. 

3. Suppose A is tall and thin [m > n). The n columns are likely to be independent. 
But if b is not in the column space, Ax = b has no solution. The least squares 
method minimizes ||6 — by solving A^Ax = A'^b. 



The Four Fundamental Subspaces 



The nullspace N{A) contains all solutions to Ax = 0. The column space C{A) 
contains all combinations of the columns. When A is m by n, N{A) is a subspace of 
R" and C{A) is a subspace of R™. 

The other two fundamental spaces come from the transpose matrix A'^ . They are 
N{A'^) and C{A'^). We caU C{A'^) the "row space of A" because the rows of A are 
the columns of A'^ . What are those spaces for our 2 by 2 example? 



A 
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transposes to A^ 
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Both columns of A'^ are in the direction of (1,2). The line of all vectors (c, 2c) is 
C(A'^) = row space of A. The nullspace of A^ is in the direction of (3, —1): 



Nullspace of A^y 



The four subspaces N{A), C{A), N{A^), C{A^) combine beautifully into the big 
picture of linear algebra. Figure A2 shows how the nullspace N (A) is perpendicular 
to the row space C{A^). Every input vector x splits into a row space part Xr and a 
nullspace part x„. Multiplying by A always(!) produces a vector in the column space. 
Multiplication goes from left to right in the picture, from x to Ax = b. 



column space 
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row space C{A^) 


nullspace N{A) ^ 


multiples of (1,2)" 


' multiples of (2, —1) 




nullspace N{A^) 
multiples of (3,-1) 



Figure A2: The four fundamental subspaces (lines) for the singular matrix A. 
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On the right side are the column space C{A} and the fourth space N{A^). Again 
they are perpendicular. The columns are multiples of (1,3) and the y's are multi- 
ples of (3, —1). If ^ were an m by n matrix, its columns would be in m-dimensional 
space R™ and so would the solutions to A^y = 0. Our singular 2 by 2 example has 
m = n = 2, and all four fundamental subspaces in Figure A2 are lines in R^. 

This figure needs more words. Each subspace contains infinitely many vectors, 
or only the zero vector x = 0. If u is in a space, so are Wu and — lOOu (and most 
importantly Ou). We measure the dimension of a space not by the number of 
vectors, which is infinite, but by the number of independent vectors. In this example 
each dimension is 1. A line has one independent vector but not two 



Dimension and Basis 



A full set of independent vectors is a "basis" for a space. This idea is important. 
The basis has as many independent vectors as possible, and their combinations fill 
the space. A basis has not too many vectors, and not too few^: 

1. The basis vectors are linearly independent. 

2. Every vector in the space is a unique combination of those basis vectors. 

Here are particular bases for R" among all the choices we could make: 

Standard basis = columns of the identity matrix 

General basis = columns of any invertible matrix 

Orthonormal basis = columns of any orthogonal matrix 

The "dimension" of the space is the number of vectors in a basis. 



Difference Matrices 
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Difference matrices with boundary conditions give exceptionally good examples of the 
four subspaces (and there is a physical meaning behind them). We choose forward 
and backward differences that produce 2 by 3 and 3 by 2 matrices: 



Forward A_|_ 
Backward — A_ 



A is imposing no boundary conditions (no rows are chopped off). Then A^ must 
impose two boundary conditions and it does: +1 disappeared in the first row and —1 
in the third row. A^w = f builds in the boundary conditions = and = 0. 

The nullspace of A contains x = (1,1,1). Every constant vector x = (c, c, c) 
solves Ax = 0, and the nullspace N{A) is a line in three-dimensional space. The 
row space of A is the plane through the rows ( — 1, 1, 0) and (0, —1, 1). Both vectors 
are perpendicular to (1,1,1) so the whole row space is perpendicular to the 
nullspace. Those two spaces are on the left side (the 3D side) of Figure A3. 
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Figure A3: Dimensions and orthogonality for any m by n matrix A of rank r. 

Figure A3 shows the Fundamental Theorem of Linear Algebra: 

1. The row space in R" and column space in R™ have the same dimension r. 

2. The nullspaces N{A) and N{A^) have dimensions n — r and m — r. 

3. -^(^) is perpendicular to the row space C{A^). 

4. N{A^) is perpendicular to the column space C{A). 

The dimension r of the column space is the "rank" of the matrix. It 

equals the number of (nonzero) pivots in elimination. The matrix has full column 
rank when r = n and the columns are linearly independent; the nuUspace only con- 
tains a; = 0. Otherwise some nonzero combination x of the columns produces Ax = 0. 

The dimension of the nuUspace is n — r. There are n unknowns in Ax = 0, 
and there are really r equations. Elimination leaves n — r columns without pivots. 
The corresponding unknowns are free (give them any values). This produces n — r 
independent solutions to Ax = 0, a basis for the nuUspace. 

A good basis makes scientific computing possible: 
1 Sines and cosines 2 Finite elements 3 Splines 4 Wavelets 
A basis of eigenvectors is often the best. 



